TAIBOM Datapack

Introduction

The TAIBOM Datapack Schema is a JSON schema designed to describe and standardise collections of datasets within the Trusted AI BOM (TAIBOM) ecosystem. This schema ensures that groups of datasets, whether used for raw data, training, or testing, are organised and verified for integrity.

Description

This schema captures essential metadata for datapacks, including:

• Name: A unique identifier for the datapack.
• Datasets: A collection of dataset objects, each containing:
  • A unique identifier (`id`) for traceability.
  • A cryptographic hash (`hash`) to ensure data integrity.

Use Case

The TAIBOM Datapack Schema is primarily used within the TAIBOM framework to:

1. Organise Dataset Collections: Provide a standardised format for grouping datasets used in AI workflows.
2. Enable Traceability: Ensure that each dataset in the collection is uniquely identified and linked to its source.
3. Streamline Dataset Verification: Facilitate verification and management of multiple datasets within a single datapack.

By adopting this schema, organisations can effectively manage and document groups of datasets, ensuring integrity and ease of use in AI system development and deployment.

---

Schemas

yaml

$id: https://github.com/nqminds/Trusted-AI-BOM/blob/main/packages/schemas/src/taibom-schemas/20-data-pack.v1.0.0.schema.yaml
$schema: https://json-schema.org/draft/2019-09/schema
title: TAIBOM Datapack
description: |
  A datapack schema that describes a collection of datasets used for raw data, training data, and testing data.  Each collection has a unique identifier and a cryptographic hash for verifying data integrity.
type: object
properties:
  name:
    type: string
    description: The name of the datapack.
  datasets:
    type: array
    items:
      type: object
      properties:
        id:
          type: string
          description: The unique identifier of the dataset, such as a VC ID or a DID.
          format: uri
        hash:
          type: string
          description: The cryptographic hash (e.g., SHA-256) of the dataset for data integrity.
      required:
        - id
        - hash
    description: |
      A list of dataset objects, each containing an ID and a cryptographic hash.
    minItems: 1
    uniqueItems: true
required:
  - name
  - datasets

json

{
  "$id": "https://github.com/nqminds/Trusted-AI-BOM/blob/main/packages/schemas/src/taibom-schemas/20-data-pack.v1.0.0.schema.yaml",
  "$schema": "https://json-schema.org/draft/2019-09/schema",
  "title": "TAIBOM Datapack",
  "description": "A datapack schema that describes a collection of datasets used for raw data, training data, and testing data.  Each collection has a unique identifier and a cryptographic hash for verifying data integrity.\n",
  "type": "object",
  "properties": {
    "name": {
      "type": "string",
      "description": "The name of the datapack."
    },
    "datasets": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "id": {
            "type": "string",
            "description": "The unique identifier of the dataset, such as a VC ID or a DID.",
            "format": "uri"
          },
          "hash": {
            "type": "string",
            "description": "The cryptographic hash (e.g., SHA-256) of the dataset for data integrity."
          }
        },
        "required": [
          "id",
          "hash"
        ]
      },
      "description": "A list of dataset objects, each containing an ID and a cryptographic hash.\n",
      "minItems": 1,
      "uniqueItems": true
    }
  },
  "required": [
    "name",
    "datasets"
  ]
}

markdown

TAIBOM Datapack

A datapack schema that describes a collection of datasets used for raw data, training data, and testing data. Each collection has a unique identifier and a cryptographic hash for verifying data integrity.

The schema defines the following properties:

name (string, required)

The name of the datapack.

datasets (array, required)

A list of dataset objects, each containing an ID and a cryptographic hash.

The object is an array with all elements of the type object.

The array object has the following properties:

id (string, required)

The unique identifier of the dataset, such as a VC ID or a DID.

hash (string, required)

The cryptographic hash (e.g., SHA-256) of the dataset for data integrity.

Additional restrictions:

• Minimum items: 1
• Unique items: true

Examples

table

name	datasets
Raw collection	[object Object],[object Object]

json

[
  {
    "name": "Raw collection",
    "datasets": [
      {
        "id": "urn:uuid:data-vc-1234",
        "hash": "3b74b7f0b762998774a97b2c4c648a0f8c51f04d6c74f2e7698f4d8b9"
      },
      {
        "id": "urn:uuid:data-vc-4567",
        "hash": "23d9f32cb1628925d3f048cf3eb8f1ea4f7e61f3f3426a0c33b5df5b5a"
      }
    ]
  }
]

Edit this schema here